Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think